Skip to content

chore(sinks): log reason for non-retryable errors - #25934

Open
petere-datadog wants to merge 3 commits into
masterfrom
peter.ehik/OPA-5715/log-reason-for-non-2xx-errors
Open

chore(sinks): log reason for non-retryable errors#25934
petere-datadog wants to merge 3 commits into
masterfrom
peter.ehik/OPA-5715/log-reason-for-non-2xx-errors

Conversation

@petere-datadog

@petere-datadog petere-datadog commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Summary

Log reason for non-retryable errors in http sinks.

Vector configuration

sources:
  dd-agent:
    type: datadog_agent
    address: 0.0.0.0:8283
    disable_logs: true
    disable_traces: true
    store_api_key: false
    split_metric_namespace: false

sinks:
  prometheus_out:
    type: prometheus_remote_write
    inputs:
      - dd-agent
    endpoint: http://localhost:9090/api/v1/write

  file_out:
    type: file
    inputs:
      - dd-agent
    path: /tmp/prometheus-metrics.log
    encoding:
      codec: json

Sample log

2026-07-30T19:10:02.173944Z ERROR sink{component_kind="sink" component_id=prometheus_out component_type=prometheus_remote_write}:request{request_id=69}: vector::sinks::util::retries: Not retriable; dropping the request. reason="Default retry strategy: Bad Request, response body: out of bounds\n"

Change Type

  • Bug fix
  • New feature
  • Dependencies
  • Non-functional (chore, refactoring, docs)
  • Performance

Is this a breaking change?

  • Yes
  • No

Does this PR include user facing changes?

  • Yes. Please add a changelog fragment based on our guidelines.
  • No. A maintainer will apply the no-changelog label to this PR.

References

Notes

  • Please read our Vector contributor resources.
  • Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
  • Some CI checks run only after we manually approve them.
    • We recommend adding a pre-push hook, please see this template.
    • Alternatively, we recommend running the following locally before pushing to the remote branch:
      • make fmt
      • make check-clippy (if there are failures it's possible some of them can be fixed with make clippy-fix)
      • make test
  • After a review is requested, please avoid force pushes to help us review incrementally.
    • Feel free to push as many commits as you want. They will be squashed into one before merging.
    • For example, you can run git merge origin master and git push.
  • If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
    run make build-licenses to regenerate the license inventory and commit the changes (if any). More details on the dd-rust-license-tool.

@petere-datadog
petere-datadog requested a review from a team as a code owner July 23, 2026 15:23
@github-actions github-actions Bot added the domain: sinks Anything related to the Vector's sinks label Jul 23, 2026
@datadog-vectordotdev

datadog-vectordotdev Bot commented Jul 23, 2026

Copy link
Copy Markdown

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: aa68c45 | Docs | Datadog PR Page | Give us feedback!

@pront pront left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Logging a response body risks disclosing sensitive data. Destinations can possibly echo data in 4xx responses, and this would create a copy of that data in Vector's error logs, potentially with broader access and longer retention than the original event data.

We already log response bodies at debug level in HttpRetryLogic. Could we do the same for this path, ideally through a shared bounded-body logging helper?

If error-level response-body logging is required for your use case, we could expose an explicit opt-in configuration field so users accept the disclosure risk.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

domain: sinks Anything related to the Vector's sinks

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants